Make MLX metallib generation automatic and stale libraries loudly detectable - #131

Incident

On 2026-07-03, text chat at temperature 0 produced incoherent multilingual gibberish with run-to-run nondeterminism on any prompt past ~1024 tokens of total context, on every MLX text model (Q35/ornith, q36-nano, and by the same mechanism Gemma4/LFM2/Psi), while short prompts stayed perfectly clean. GGUF/llama.cpp models were unaffected.

Root cause

swift build never compiles mlx-swift's Metal kernels — there is no metallib rule in the SwiftPM manifest — yet the MLX runtime hard-requires default.metallib and loads whatever file it finds by name, with no version validation. The repo has vendored a copy (vendor/mlx-swift_Cmlx.bundle, tracked in git and consulted first by both the CLI bundle resolver and the test harness) that was built from pre-0.30 mlx sources at the initial public release and never regenerated.

mlx 0.31.x rewrote the 2-pass sdpa_vector kernel ABI (ml-explore/mlx#3023: bf16 partials instead of fp32, gqa_factor buffer removed, new blocks function constant). The 1-pass kernel ABI didn't change — so everything below 1024 keys worked — but the moment single-token decode crossed the k_len >= 1024 two-pass dispatch threshold ('d'/'s'-class Apple GPUs, e.g. M4 Max), the 0.31.1 host code drove pre-0.30 kernel binaries: NaN/garbage partials, nondeterministic output. Isolated probe on the exact decode shape (q [1,16,1,256] bf16, GQA 16:2, cache-style strided k/v views): clean at N=1023, ~1e34/NaN and run-to-run drift at N≥1024 with the stale lib; drift 0.0 and ref error ≤2e-4 at all N with a freshly compiled one.

Skip-if-exists copy/symlink logic in MLXBundleSupport, MLXTestSupport, and install.sh then propagated the stale blob into every build dir, install, xctest bundle, and release staging tree indefinitely.

Changes

• scripts/build_mlx_metallib.sh — compiles the AOT .metal sources from the current mlx-swift checkout with mlx's mandatory -fno-fast-math flags, links default.metallib, and writes a default.metallib.version sidecar (mlx core version, mlx-swift pin revision, kernel-source SHA-256). Installs into .build (both configs) and refreshes the tracked vendor bundle. --verify-only <bundle> exits non-zero on any drift from the checkout.
• vendor/mlx-swift_Cmlx.bundle — the tracked metallib regenerated from mlx-swift 0.31.4 (mlx core 0.31.1), now with its version stamp.
• Runtime validation — MLXRuntimeVersion (MereRunCore) exposes the compiled-in mlx core version via mlx-c; MLXBundleSupport.ensureAvailable compares it against the bundle stamp at startup: hard error on mismatch (override: MERERUN_ALLOW_METALLIB_MISMATCH=1), loud warning when unstamped, self-heal by preferring stamp-matched candidate bundles, atomic always-fresh compatibility copies (previously skip-if-exists, which let stale flat copies outlive bundle updates), and a bundle husk without a metallib no longer counts as present.
• MLXTestSupport — replaces stale colocated symlinks/copies instead of skipping them. A leftover absolute symlink into another checkout's vendored bundle had pinned test runs to the broken metallib; steady-state (symlink already pointing at the current source) remains a no-op.
• SDPAVectorKernelTests — GPU regression test on the exact failing shape across the 1024-key dispatch boundary (1023 control / 1100 / 3300): unfused-fp32 agreement plus bit-identical determinism over 3 runs. Fails against a stale metallib (verified live against two independent stale copies), passes against a stamped fresh one. Run with MERERUN_TEST_MLX_DEVICE=gpu.
• Packaging — build_mere_run_app.sh rebuilds + stamps the metallib after swift build and refuses to package a bundle failing --verify-only; install.sh carries the stamp alongside the compatibility copies and warns when installing an unstamped library.

Verification

• All four runtime validation paths exercised on a real binary: matched (silent), mismatched (hard error naming both versions), override (warning + run), unstamped (loud warning).
• Original failing repro (3.3k-token prompt, ornith-35b-mlx, temp 0): run-to-run identical, coherent output — release CLI and the refreshed installed CLI (via q36-nano) both verified.
• MERERUN_TEST_MLX_DEVICE=gpu swift test --filter "SDPAVectorKernelTests|Gemma4DecodeFusedKernelsTests": 6 tests, 0 failures.
• --verify-only green on the committed vendor bundle; red on a tampered stamp.

Notes for reviewers

• The vendor metallib is a binary diff (3.83 MB → 2.86 MB); the sidecar records exactly what it was built from. It must be regenerated (script does it automatically) on every mlx-swift bump — the new runtime check turns forgetting into a hard error instead of silent corruption.
• The release-tools packaging pipeline stages its own vendored bundle copies and should adopt build_mlx_metallib.sh --verify-only; that lives in the private repo and is not covered here.
• Pre-existing and unrelated: swift test -c release fails to build (seedLoadedModelForTesting is debug-only).
• CI suggestion: run the regression test on an Apple Silicon runner with MERERUN_TEST_MLX_DEVICE=gpu.

🤖 Generated with Claude Code